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Abstract: Web resources allow a learner to have more opportunities for study at any time and any place. It is still difficult, 
however, for learners to choose the right study materials to match their desired learning. A competence-based system for 
recommending study materials from the Web (COSREW) is proposed, based on the learner's competences. COSREW 
generates a list of learning paths, and extracts search terms from the competence statements on the chosen learning path. 
Three experiments were conducted to evaluate COSREW's recommendations. The first explored the differences between 
search engines and the qualities of the study material links in helping learners achieve their competences. The second 
experiment explored the differences between search keywords, and the third experiment compared COSREW with freely- 
browsing learning modes. The results showed that the Web is currently not a good resource for a pedagogically informed 
competence-based system, since Web pages predominantly comprise text-based subject matter content with little support 
for learning competence or capability. 

Keywords: Competency Model; Competence Structure; Web-based Learning; Internet Supported Learning; Pedagogy; Self¬ 
learning 

1 Introduction 

The aim of this paper is to propose a competence-based system for recommending study materials from the 
Web (COSREW) which provides links as appropriate study materials. The main objective of COSREW is to help 
learners find study materials from the Web as supplementary resources outside the classroom. The Web can 
be an effective learning resource since it seems to offer diverse learning materials which could address learner 
needs. COSREW is based upon the COMBA competency model (Sitthisak et al. 2008), where a competence is 
conceptualised as consisting of three major components: subject matter, capability, and context. Within 
COSREW, the learner's existing and desired competences lead to a selection of different learning paths. 
Keywords are extracted from the competence statements on the chosen learning path, and the resulting links 
are presented to the learner as recommendations. COSREW deploys an XML-schema which provides a 
common framework for abstracting the information in a competence structure. It can be reused for any 
knowledge domains of subject matter content. 

In the following sections, the major literature on competency models and competence structures is presented, 
followed by the COSREW design. Later, a method of implementing a competence structure and its XML- 
schema is proposed. Next, three experimental studies which evaluated COSREW's recommendations are 
presented and the results discussed. Finally, conclusions and further studies are addressed. 

2 Competency Model and Competence Structure 

This section introduces a competency model based on intended learning outcomes and discusses examples of 
existing competency models. 

2.1 Definition of Competency 

There are many definitions of competency in the literature. According to Smith (1996), 'competency' refers to 
the ability to do a particular activity to a prescribed standard. A definition of competency is given by the HR- 
XML consortium (HR-XML 2004): 
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"specific, identifiable, definable, and measurable knowledge, skill, ability and/or other deployment-related 
characteristic (e.g. attitude, behaviour, physical ability) which a human resource may possess and which is 
necessary for, or material to, the performance of an activity within a specific business context." 

Another definition is given by McClelland (1973), "competency can be the knowledge, skills, traits, attitudes, 
self-concepts, values, or motives related to job performance or important life outcomes." Friesen and 
Anderson (2004) define a competency as "the integrated application of knowledge, skills, values, experience, 
contacts, external knowledge resources and tools to solve a problem, to perform an activity or to handle a 
situation." Finally, Cheetham and Chivers (2005) suggest the following general definition of competence: 
"effective overall performance within an occupation, which may range from the basic level of proficiency 
through the highest levels of excellence." 

2.2 Existing Competency Models and Structures 

There are two existing international competency standards, IMS RDCEO (IMS RDCEO 2002) and FIR-XML (FIR- 
XML 2004). Their data models or schemas are minimalist but extensible. 

IMS RDCEO provides five elements in its information model: identifier, title, description, definition and 
metadata. There are some disadvantages to this competency model, however, such as the oversimplification 
of the concept of competency, and the lack of provision for an adequate semantic level to support intelligent 
decisions. In particular, the model does not take into consideration explicitly important elements such as the 
knowledge and skills of learners (Baldiris et al. 2007), nor does it support a common language of competency. 

FIR-XML consortium's competency schema has nine components: name, description, required, competency Id, 
Taxonomy Id, Competency Evidence, Competency Weight, Competency, and user Area. FIR-XML competency 
can refer to knowledge, skill, ability, attitude, behaviour, or a physical ability. In terms of its implementation, it 
aims to be used by different people within different disciplines such as human resources management, 
industrial psychology, and education 

Sampson and Fytros (2008) identify some drawbacks to these competency standards, such as the titles and 
descriptor elements in these models not being directly machine understandable. Moreover, both standards 
adopt a competence description but do not take a proficiency level into consideration, although this is 
important to the competency concept (Sampson and Fytros 2008). Their proposed competency model is 
shown in Figure 1. 



Figure 1: Competency Model Elements (Sampson and Fytros 2008) 

The proficiency level in this competency model refers to skills, knowledge, and attitudes, but it remains 
inadequate. Proficiency can refer to either skills or knowledge in the model, but this is incompatible with 
considering an intended learning outcome as the combination of capability (skill) and subject matter 
(knowledge). 

The competence structure specifies the range of competence elements/nodes for a particular knowledge 
domain and highlights the relationship between competence nodes. There are some existing competence 
structures which were designed from different aspects of competence. One competence structure was 
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developed by Kickmeier-Rust et al. (2006) as shown in Figure 2. One node represents a competence state 
which is a set of all available competencies of a person. The prerequisite relationships are defined within this 
set of competencies. Each competency in a state represents a problem or subject matter which a learner is 
required to solve. 



Figure 2: Competence Structure Established by the Prerequisite Function (Kickmeier-Rust et al. 2006) 

Another competence structure was proposed by Heller et al. (2006). Flowever, this structure represents a 
competence-based knowledge structure. It is extended from a knowledge structure as is shown in Figure 3. 
They introduced two other sets of learning objects (LOs) and related skills for solving problems corresponding 
to each node within the structure. Nonetheless, this structure is based on knowledge representation. 



Figure 3: Overview of Knowledge Structure of Domain Q = 'a, b, c, d, e' (Heller et al. 2006) 

2.3 COMBA model 

The proposed model for this research draws on the multidimensional competency model (called COMBA) 
proposed by Sitthisak et al. (2008). This considers the learners' learnt capability instead of their knowledge 
level and views competences and learnt capabilities as a multidimensional space (Sitthisak et al. 2008). The 
COMBA model (Figure 4) consists of three major components: subject matter, capability, and context. 
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Figure 4: Competency Model Derived from COMBA Model Proposed By Sitthisak et al (2008) 

The competence structure, which is designed based on the COMBA model, contains nodes comprising 
capability, subject matter, and context (optional).There are some existing competence structures based on 
COMBA model. One sample is a tree of nursing competencies from the UK Royal College of Nursing introduced 
by Sitthisak et al (2009). The general form of this competence structure is shown in Figure 5. The relationship 
between nodes is parent-child with no ordering on the same level. A parent-child relationship identifies what 
the learner must be able to do (child) before something else (parent) can be learned. 


B 



Figure 5: Nursing Competency Tree from the UK Royal College of Nursing (Sitthisak et al. 2009) 

Another competence structure was developed by Iskandar et al. (2010). This competence structure is shown in 
Figure 6. The competence nodes in this structure are all intended learning outcomes independent of context. 
This structure is based on the sport of rowing in the motor skill domain. The relationship between the 
competence nodes is parent-child (as in the previous competence structure example). 



Figure 6: Conceptual Model of Learning Outcomes in the Motor Skill Domain (P Iskandar et al. 2010) 

3 COSREW design 

This section describes the application of the COMBA model to the COSREW design, and presents the system 
process of suggesting study materials from the Web. 
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3.1 Apply the COMBA Competency Model 

There are some reasons why a COMBA competency model should be considered in this study. 

First is the issue of a machine-processable, sharable, and modifiable representation of learner competence. 
Each individual learner's competences are clearly defined with a competency model. A learner's competences 
are connected to prerequisite or enabling competences and are formed as a network structure. 

Second is the navigation of a competence structure or network. Navigating the structure offers various routes 
for providing learners with study material links to enable them to achieve a learning outcome. 

The third issue is identifying the context of a learner's competence. Learners may have differing levels of 
proficiency in relation to a given intended learning outcome, depending upon the context. The defined context 
of a competence distinguishes a competence from an intended learning outcome. 

The fourth issue is that the COMBA model formally defines the combination of 'capability' and 'subject matter' 
as 'intended learning outcome'. Intended learning outcomes describe what learners need to be able to do to 
complete a course satisfactorily (Macdonald 1999). An intended learning outcome (previously also known as 
an educational objective) has long been a central component in the design and structure of educational and 
training systems, particularly in schools and in industrial training (Reigeluth 1999, Gagne et al. 2004).In 
addition, an intended learning outcome provides a clear expectation on the part of the student which is a 
crucial part of their effective learning (Ramsden 1992). 

3.2 COSREW Processes 

The overview of the process within COSREW, illustrated at Figure 7, shows how COSREW deals with learners' 
competences and how it recommends appropriate study material links from the Web to learners so that 
learners can achieve their intended learning outcomes. 

Subject - k Learner’s competences - k Learning .- h K . .- K Study Materials 

Matter -/ (Desired & Existing) - v Paths * y r Links 



Figure 7: Process within COSREW 


Indicate the steps of the process within the system 


First, a sub-process constructs a learner's competence structure so that COSREW can generate lists of targeted 
subject matter and competences for learners to choose from. After the chosen subject matter and 
competences (desired and existing) are obtained, COSREW then generates a list of learning paths. COSREW 
constructs a search based on the chosen learning paths, and then suggests the resulting links to learners. 

The reason for considering competence statements as the source of keywords for a search engine is explained 
as follows. Designing a system that will enable a learner to find appropriate study materials from the Web 
without any interaction from a teacher requires a method of obtaining information on the Web. Gordon and 
Pathak (1999) discussed four different methods for locating information on the Web: 

• Go directly to a webpage location. 

• Hypertext links emanating from a webpage provide built-in associations to other pages. 

• Narrowcast services can push pages that meet particular user profiles. 

• Use search engines to find and then furnish information on the Web that hopefully relates to that 
description. 

As the learner's competences are the information (input) that the system obtains from the learners, so the 
system should recommend study materials based on these competences. The appropriate method is to use 
queries to a search engine based on the desired competences. There are many search engines, for example, 
Google, Bing, Yahoo, Alta Vista and Microsoft. The type of search engine is not considered as an important 
point in this research: the search engine is used merely as the intermediate tool to get relevant study 
materials, using the learners' competences. 
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4 Design of Competence Structure 

This section begins with a procedure for designing a competence structure from an existing course syllabus. 
The XML-schema of the resulting competence structure is then presented which may be useful for 
construction, implementation, and evaluation, especially for those approaches which require the advantages 
of usability, semantics, and modifiability. 

4.1 Method of designing a Competence Structure 

In order to design a structure of competence information on the intended learning outcomes for the specific 
subject matter content of a course, the topic, syllabus, or curriculum is required. The subject matter content is 
categorised and tagged with relevant learner capabilities and contexts in order to derive the structure of 
competences. 

4.1.1 Step 1: Choose the topic 

Intended learning outcomes in UK education are published by, for example, AQA, OCR, and Edexcel. In this 
research, the intended learning outcomes from the course specification for photosynthesis at a Key Stage 4 
(GCSE, UK) from AQA (revised version), (The Assessment and Qualifications Alliance 2010) were chosen for 
constructing the competence structure. Examples of intended learning outcomes from this specification are as 
follows: 

■ recall photosynthesis equation 

■ recall photosynthesis definition 

■ define chlorophyll 

■ interpret data showing how factors affect the rate of photosynthesis 

■ demonstrate a photosynthesis procedure 

■ predict the rate of photosynthesis in different conditions using computer simulations 

4.1.2 Step 2: Task Analysis of Subject Matter 

Next, the subject matter is extracted from all the intended learning outcomes and categorized into the four 
types of content following Merrill's CDT (Component Display Theory) analysis (Merrill 1994). They are: fact, 
concept, procedure and principles. A specific technical definition of each category is provided in (Gilbert and 
Gale 2008). 

Table 1: Definition of CDT Categories of Subject Matter (Gilbert and Gale 2008) 


CDT Categories 

Definition 

Fact 

Fact pair 

Concept 

Name of concept 

Superordinate concept class 

Attribute-value pairs 
that classify objects 

Procedure 

Name of procedure 
used in situation 
to achieve a goal 
via a set of steps 
using tools 

Principle 

Name of principle 
applied in situation 
involves cause-effect relationships 
between objects or events. 


For photosynthesis at Key Stage 4, the list of subject matter and their CDT categorizations are provided in 
Table 2. 
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Table 2: Subject Matter Content and Categorization of Photosynthesis for Key Stage 4 Learners 


CDT Subject Matter Category 

Subject Matter 

Fact 

Photosynthesis equation, photosynthesis definition, substance, energy, sun, bulb, 
gas, C0 2 , H 2 0, 0 2 , plant cell, location, mesophyll cell, etc. 

Concept 

Chlorophyll, light, carbon dioxide, water, oxygen, chloroplast, etc. 

Procedure 

Photosynthesis procedure 

Principle 

Photosynthesis rate 


Task analysis in instructional design is a process of analyzing and expressing the nature of learning content so 
that a learner knows how to perform (knows what to do) (Jonassen et al. 1998). The intention is to represent 
the subject matter content in the form of a diagram which is based on CDT categories. The CDT category "fact" 
comprises two elements, making a 'fact pair'. Notationally, each element can be represented by a circle. For 
example, the fact, 'Chemical formula of Carbon Dioxide is C0 2 ' is represented by a fact pair, 'chemical formula' 
and 'C0 2 ' as shown in Figure 8. 


Chemical formula of Carbon Dioxide is C02 


chemical formula ^ 

) C02 


Figure 8: Task Analysis of Fact 'Chemical formula of Carbon Dioxide is C0 2 ' 

The CDT category "concept" involves the concept name and its super ordinate class, which is normally a fact 
pair. The relationship between class and super ordinate class is 'is a kind of. Notationally, a concept may be 
represented by a triangle. A concept comprises a number of attribute-value pairs which distinguish the 
concept from other similar concepts. For example, the concept of carbon dioxide is shown in Figure 9. 



Figure 9: Task Analysis of the Concept of Carbon Dioxide 

The CDT category "procedure" consists of a set of steps. Notationally, each step may be represented by a 
square. For example, the photosynthesis procedure is shown inFigurelO. 
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Photosynthesis procedure 


Chlorophyll absorbs light energy 


Light energy converts carbon 
dioxide and water 


Oxygen and sugar are released 


Figure 10: Task Analysis of the Photosynthesis Procedure 


The CDT category "principle" involves causes and their effects. Notationally, the principle itself may be 
represented by a pentagon. For example, the principle of photosynthesis rate is shown in Figurell. Causes are 
shown on the left side of pentagon and the right side shows the effects or results. 



Figure 11: Task Analysis of the Principle of Photosynthesis Rate 

4.1.3 Step 3: Decompose Levels of Task Analyses 

Step 3 composes of 3 sub steps as follows: 

• Sub-Step 1 (Construct Level 0): Identify the topic of the whole subject or syllabus. For example, level 
0 contains one subject matter as 'photosynthesis'. 

• Sub-Step 2 (Construct Level 1): From the course specification structure and its top level statements, 
identify the subject matter which comprises the level 0 topic. For example, from the AQA Key Stage 4 
course specification, level 1 comprises four subjects as 'photosynthesis definition', 'photosynthesis 
equation', 'photosynthesis procedure', and 'photosynthesis rate'. 

• Sub-Step 3 (Construct Level i+1): Consider each of the subject matters identified in level i. For any 
concepts, procedures, or principles, place their constituent elements (which may be facts or further 
concepts, procedures, and principles) at this level i+1. These constituent elements are those as found 
in sub-step 2. If there are no such subject matters, then END. Otherwise, repeat this sub-step 3 for 
level i+2. 

4.1.4 Step 4: Structure Subject Matter Relationships 

Assign a parent-child relationship between subject matter nodes as follows: 

• Sub-Step 1: The level 0 subject matter is the parent of level 1 subject matter. 

• Sub-Step 2: Each level i+1 subject matter is the parent of its level i+2 constituent elements. 

For example, the resulting structure of the Key Stage 4 photosynthesis subject matter is illustrated in Figurel2. 
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4.1.5 Step 5: Tag Each Node with Capability and Context 

In order to develop a competence structure, each node of subject matter requires tagging with a 
corresponding capability and a context. These tags are taken from the course specification. For example, the 
resulting competence structure of photosynthesis for Key Stage 4 learners is shown in Figurel3. 

The competence structure is represented as a tree structure and the relationships between nodes are 
represented as parent-child relationships. This method of designing a competence structure allows developers 
to design the competence structures they may need from existing course specifications, curricula, and syllabi. 
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Figure 12: Knowledge Structure of Photosynthesis for Key Stage 4 Learners 
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Figure 13: Competence Structure of Photosynthesis for Key Stage 4 Learners Domain Mapped XML-Schema 


www.ejel.org 


1 


ISSN 1479-439X 














The XML format for representing a competence structure is now illustrated. XML enables a focus upon the 
definition of shared vocabularies for exchanging information, supporting the reuse of the content in other 
applications (St.Laurent 1998). In this section, the derivation of the XML schema is described, based upon an 
entity relationship diagram. 

4.1.6 Entity Relationship Diagram Representing Competence Structure 

The entity relationship diagram (ERD) of Figure 14 generally represents all the data entities and attributes in 
any competence structure, including intended learning outcomes, different types of subject matter content 
(including their task analyses), and contexts. 



Figure 14: ER Diagram of Competence Structure 


4.1.7 Mapping XML-Schema 

An XML-Schema defines the terms, relationships, and constraints required to support communication in a 
particular application domain (Carlson 2001). All schemas provide some degree of definition and 
documentation for an XML vocabulary. For a competence-based system, an XML-schema represents a 
common framework for abstracting the required information. Figure 15 illustrates an XML-schema for a 
competence structure. This schema is the ER diagram of section 4.2.2 represented in XML. 
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Figure 15: Mapped 

5 Experiment 

In this experiment, the search results given by the Google browser (www.google.com) were compared to those 
given by the API to Google, in terms of user ratings of achievement. The Google API is the mechanism by which 
COSREW carries out a search. 

In general, the Google browser tends to weight the search results based upon the user's browsing activities, 
but this information is not employed by the Google API (Kilgarriff 2007). Hence, the results from their searches 
can be different. 

5.1 Experimental Methodology 

The research question of experiment one concerned the users' reactions (Kirkpatrick 2007) and required 
expert review. The topic was photosynthesis for Key Stage 4 learners. Four competence nodes were used, 
representing a range of competence types on Bloom's taxonomy from 'recall' to 'predict', as follows: 

Recall a photosynthesis equation 

Demonstrate a day time photosynthesis procedure 

Predict a photosynthesis rate 

Define chlorophyll 

Keywords (combination of capability, subject matter, and context) were generated for each of the nodes. For 
example, the node "Predict a photosynthesis rate" yielded the keywords "predict", "photosynthesis", and 
"rate". These keywords were submitted to Google (www.google.com) on the one hand, and to the Google API 
on the other. The first three resulting links were presented to the participant, who rated them. 

The estimated number of participants required was obtained using G*Power software (Buchner et al. 2010) 
which was designed as a general stand-alone power analysis program for statistical tests commonly used in 
social and behavioural research (Faul et al. 2007). The number of participants required in each experiment 
differed according to the nature of the experiment. The required sample size (ie the required number of links) 
of this experiment was 24 according to G*power, using an effect size f = 1, an alpha error probability = 0.05, 
power = 0.8, the test family as F-test, the number of groups = 2, and the statistical test as ANOVA fixed effects. 


<!- declare all types of table-> 

<xsd:complexType name="UserInfoType"> 
<xsd:sequence> 

<xsd:element name="User_ID" type="xsd:string"/> 


</xsd:sequence> 

</xsd:complexType> 

<!-- Content of all elements / all tables --> 

<xsd:element name="Competence_Data"> 

<xsd:complexType> 

<xsd:sequence> 

<xsd:element name="UserInfo" type="UserInfoType" minOccurs="0" maxOccurs="unbounded"/> 


</xsd:sequence> 

</xsd:complexType> 

<!-- Declare Primary keys and other keys--> 

<xsd:key name="PK_UserInfo_User_ID"> 
<xsd:selector xpath=".//UserInfo7> 
<xsd:field xpath="User_ID"/> 
</xsd:key> 


<!-- declare foreign keys --> 

<xsd:keyrefname="FK_UserInfoUserDComp" refer="PK_UserInfo_User_ID"> 
<xsd:selector xpath=".//UserDComp"/> 

<xsd:field xpath="User_ID"/> 

</xsd:keyref> 


XML-Schema of Competence Structure 

One 
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5.2 Questionnaire Analysis 

The participant gave a rating of 'helpfulness to achieve learning outcome' to each link on a scale of six (1, 2 , 3, 
4, 5 , and 6). The 6-point Likert scale was adopted in order to avoid a neutral or mid-point, resulting in a 'forced 
choice'. Eliminating the mid-point category from the Likert scale reduces social desirability bias (Garland 1991). 
Chen, Lee, & Stevenson (1995) and Stening & Everett (1984) found that Asian participants were more likely to 
choose the midpoint of Likert scale item than Americans. The use of forced choice in the Likert scale, as is 
common in opinion research, was considered appropriate given the Thai participants in the experiment. 

In this experiment, the participant was obliged to rate the links as either non-useful (1-3) or useful (4-6). The 
scales were: 

• This website is not related to any materials required in order to learn how to achieve a competence 

• This website gives little information required in order to learn how to achieve a competence 

• This website gives some information required in order to learn how to achieve a competence 

• This website gives useful information required in order to learn how to achieve a competence 

• This website gives very useful information required in order to learn how to achieve a competence 

• This website gives, not only the very useful information required in order to learn how to achieve a 
competence, but also with systematic feedback 

5.3 Experimental Results 

A two-way ANOVA was used to analyse the obtained data in order to find any significant differences between 
mean ratings of the search engine types and competence node types. 

'Search Engine_Type' comprised two levels: Google and Google API. 'Competence_Node' type comprised four 
levels: 'Recall a photosynthesis equation', 'Demonstrate a day time photosynthesis procedure', 'Predict a 
photosynthesis rate', and 'Define chlorophyll'. 

Table 3 and Table 4 show the descriptive statistics, and tests of between-subjects effects respectively. 

Table 3: Means and Standard Deviations of Ratings of Links (Google VS. GoogleAPI) 


SearchEngine Type 

Competence Node 

Mean 

Std. Dev 

N 

Google 

Recall a photosynthesis equation 

4.0 

2.00 

3 


Demonstrate a day time photosynthesis procedure 

2.3 

0.58 

3 


Predict a photosynthesis rate 

1.0 

0.00 

3 


Define chlorophyll 

1.3 

0.58 

3 


Total 

2.2 

1.39 

12 

GoogleAPI 

Recall a photosynthesis equation 

4.3 

1.53 

3 


Demonstrate a day time photosynthesis procedure 

2.3 

0.58 

3 


Predict a photosynthesis rate 

1.0 

0.00 

3 


Define chlorophyll 

1.3 

0.58 

3 


Total 

2.3 

1.55 

12 

Total 

Recall a photosynthesis equation 

4.2 

1.60 

6 


Demonstrate a day time photosynthesis procedure 

2.3 

0.52 

6 


Predict a photosynthesis rate 

1.0 

0.00 

6 


Define chlorophyll 

1.3 

0.52 

6 


Total 

2.2 

1.50 

24 


Table 4: Tests of Between-Subjects Effects (Google vs GoogleAPI) 


Source 

Type III Sum of Squares 

Df 

Mean Square 

F 

Sig. 

SearchEngine_Type 

0.04 

1 

0.04 

0.04 

0.837 

Competence_l\lode 

36.46 

3 

12.15 

12.68 

0.000 

SearchEngine_Type * Competence_Node 

0.13 

3 

0.04 

0.04 

0.987 

Error 

15.33 

16 

0.96 



Total 

169.00 

24 
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The data obtained (as shown in Table 3) gave the statistical results (Table 4) as follows. 

1. There was no significant interaction effect between type of search engine and type of competence node 
(0.05<p = 0.987). 

2. There was no significant main effect for type of search engine. This indicates that there was no significant 
difference between the means of ratings of links generated from Google and GoogleAPI (0.05< p = 0.837) 

3. There was a significant main effect for type of competence node. This indicates that there were significant 
differences among mean ratings of links based on different types of competence node (p <0.05). 

6 Experiment Two 

Experiment two was conducted to explore whether the search results for competence keywords were better 
than those for subject keywords in terms of learner achievement of the competence concerned. The research 
question was whether the Web links from competence keywords were likely to be more helpful for a learner 
than those from subject keywords. 

6.1 Experimental Methodology 

The methodology was similar to experiment one. While experiment one considered 'types of search engine', 
experiment two considered 'types of keywords'. There were two types of keywords: subject (only subject 
matter) and competence (combination of capability, subject matter, and context). 

6.2 Questionnaire Analysis 

The questionnaire was similar to experiment one (see section6.2), where the participant gave a rating of 
'helpfulness to achieve learning outcome' to each link. 

6.3 Experimental Results 

A two-way ANOVA was used to analyse the obtained data in order to find any significant differences between 
mean ratings of the keywords types and competence node types. 

'Keyword_Type' comprised two levels: subject and competence. 'Competence_Node' comprised four levels: 
'Recall a photosynthesis equation', 'Demonstrate a daytime photosynthesis procedure', 'Predict a 

photosynthesis rate' and 'Define chlorophyll'. Table 5 and Table 6 show descriptive statistics and the tests of 
between-subjects effects respectively. 

Table 5: Means and Standard Deviations of Ratings of Links (Subject vs Competence) 


Keyword Type 

Competence Node 

Mean 

Std. Deviation 

N 

Subject 

Recall a photosynthesis equation 

5.0 

0.00 

3 


Demonstrate a day time photosynthesis procedure 

2.3 

0.58 

3 


Predict a photosynthesis rate 

4.7 

0.58 

3 


Define chlorophyll 

4.7 

0.58 

3 


Total 

4.2 

1.19 

12 

Competence 

Recall a photosynthesis equation 

4.7 

0.58 

3 


Demonstrate a day time photosynthesis procedure 

1.7 

0.58 

3 


Predict a photosynthesis rate 

5.0 

0.00 

3 


Define chlorophyll 

5.0 

0.00 

3 


Total 

4.1 

1.50 

12 

Total 

Recall a photosynthesis equation 

4.8 

0.41 

6 


Demonstrate a day time photosynthesis procedure 

2.0 

0.63 

6 


Predict a photosynthesis rate 

4.8 

0.41 

6 


Define chlorophyll 

4.8 

0.41 

6 


Total 

4.1 

1.33 

24 


Table 6: Tests of Between-Subjects Effects (Subject vs Competence) 


Source 

Type III Sum of Squares 

Df 

Mean Square 

F 

Sig. 

Keyword_Type 

0.42 

1 

0.42 

0.20 

0.661 

Competence_Node 

36.13 

3 

12.04 

57.80 

0.000 

Keyword_Type * Competence_Node 

1.13 

3 

0.38 

1.80 

0.188 

Error 

3.33 

16 

0.21 



Total 

449.00 

24 





The data obtained (as shown in Table 5) gave the statistical results (Table 6) as follows. 
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1. There was no significant interaction effect between types of keywords and types of competence node 
(0.05<p = 0.188). 

2. There was no significant main effect for types of keywords. This indicates that there was no significant 
difference between the means of ratings of links generated from subject and competence keywords 
(0.05<p = 0.661) 

3. There was a significant main effect for types of competence node. This indicates that there were 
significant differences among mean ratings of links based on different types of competence node (p 
<0.05). 

7 Experiment Three 

Experiment three was conducted to compare whether learning using COSREW was better than learning by 

freely browsing. 

7.1 Experimental Methodology 

The pictorial representations of both learning modes are shown in Figure 16 and Figure 17. 


Desired Competence 


Existing Competence 




Competence Gaps 


I \ Learning Paths I \ Keywords I \ I ^ Links 


Figure 16: COSREW Learning Mode 


Keywords 

Figure 17: Freely-browsing Learning Mode 


_k Search 

- v Engine 


I S Links 


In the COSREW learning mode, learners were given a set of subject matter to study, and they decided their 
own choice of competences. COSREW generated the keywords from the chosen competences and suggested 
links to learners. In the freely-browsing learning mode, learners were given the same set of subject matter to 
study, and were required to decide the keywords to be given to the Google search engine on their own. 

The experiment was concerned with the second, 'learning', level of Kirkpatrick's four levels of evaluation 
(Kirkpatrick 2007). The participants were assigned to one of two groups: one group experienced the COSREW 
learning mode and the other group experienced the freely-browsing learning mode. All participants were 
required to take a pre-test and a post-test, before and after experiencing the respective learning modes. The 
pre-test and post-test were the same for all participants, being a multiple choice test consisting of 10 
questions. The scores obtained from the pre-test and post-test were compared for each learning mode. 

The required sample size of this experiment was 6 according to G*power, using an effect size f = 1, an alpha 
error probability = 0.05, power = 0.8, the test family as F-test, the number of groups = 2, and the statistical test 
as ANOVA repeated measures, within-between interaction. 


7.2 Questionnaire Analysis 

The questions in the pre-test/post-test were based on selected subject matter of the Key Stage 4 
photosynthesis competence structure (see Figurel3). The chosen subject matter was as follows: 

Photosynthesis rate 
Photosynthesis procedure 
Chlorophyll 

Carbon dioxide, Oxygen, Water, Glucose 
Chloroplast 

Table 7 shows some of the pre-test/post-test questions. 
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Table 7: Examples of Questions in Pre-test and Post-test (Experiment III) 


Subject Matter Content 

Questions in Pre-Test/Post-Test 

Photosynthesis rate 

Which factor does not affect the rate of 
photosynthesis? 

Photosynthesis procedure 

What are the products of photosynthesis? 

Chlorophyll 

Which cells in leaf contain chlorophyll? 

Carbon dioxide, Oxygen, Water, Glucose 

Which chemical formulas represent carbon dioxide, 
oxygen, water and glucose respectively? 

Chloroplast 

A key molecule NOT found in a chloroplast is ... 


7.3 Experimental Results 

Two-way repeated measures ANOVA was used to analyze the obtained test scores, in order to determine the 
better learning mode. 'Learning mode' comprised two levels, freely-browsing and COSREW. 'Test type' 
comprised two levels, pre-test and post-test. 


Table 8, Table 9, and Table 10 show the descriptive statistics, the tests of within-subjects effects and the tests 
of between-subjects effects. Figure 18 displays the profile graphs. 

Table 8: Mean and Standard Deviation of Test Scores 


Test_Type 

Learning_Mode 

Mean 

Std. Deviation 

N 

Pre_Test 

Freely_Browsing 

7.0 

1.41 

4 


COSREW 

4.5 

1.29 

4 


Total 

5.8 

1.83 

8 

Post_Test 

Freely_Browsing 

7.8 

1.89 

4 


COSREW 

6.3 

1.26 

4 


Total 

7.0 

1.69 

8 

Total 

Freely_Browsing 

7.4 

1.50 

8 


COSREW 

5.4 

1.41 

8 


Total 

6.4 

1.76 

16 



Learning Mode 


Freeiy.Browsing 


COSREW 


Figure 18: Profile Graph of Mean Ratings of Test Scores of Pre-Test and Post-Test for Two Learning Modes 
(Error Bars Show +- 1SE) 
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Table 9: Tests of Within-Subjects Effects 


Source 

Type III Sum of Squares 

df 

Mean Square 

F 

Sig. 

Test_Type 

6.25 

1 

6.25 

4.84 

0.07 

Test Type * Learning Mode 

1.00 

1 

1.00 

0.77 

0.41 

Error 

7.75 

6 

1.29 




Table 10: Tests of Between-Subjects Effects 


Source 

Type III Sum of Squares 

df 

Mean Square 

F 

Sig. 

Learning_Mode 

16.00 

1 

16.00 

5.12 

0.06 

Error 

18.75 

6 

3.13 




The data obtained (as shown in Table 8) gave the statistical results (Table 9 and Table 10) as follows. 

• There was no significant interaction effect between test type and learning mode (0.05< p = 0.41). 

• Accepting alpha = 0.10 as a "suggestive" level of significance, there was a significant test type effect. The 
result suggested that there was a significant difference between the mean test scores between pre-test 
and post-test at a significance level of 0.10 (0.05< p = 0.07 <0.10). Inspection of the means in Table 8 
shows that the post-test mean was significantly higher than the pre-test mean. 

• Accepting alpha = 0.10 as a "suggestive" level of significance, there was a significant learning mode effect. 
The result suggested that there was a significant difference between the mean test scores between a 
freely-browsing learning mode and a COSREW learning mode (0.05< p = 0.06 <0.10). Inspection of the 
means in Table 8 shows that the mean of the freely-browsing learning mode was significantly higher than 
the mean of the COSREW learning mode. 

The profile plot illustrates the interaction (Figure 18). The profile lines are apparently not parallel, visually 
suggesting an interaction, but this was not a statistically significant effect. 

8 Discussion 

8.1 Experiment One 

Experiment one sought to determine whether the search results given by the Google browser 
(www.google.com) were significantly better than those given by the Google API in terms of their rating of 
'helpfulness to achieve learning outcome'. The means ratings of links between the two types of search engine 
did not differ according to type of competence node; the two types of search engine gave similar results. This 
suggests that future development of search-based learning materials does not need to be concerned about 
any differences in the resulting links between the search engine itself or the provided API. 

8.2 Experiment Two 

Experiment two sought to determine whether the search result ratings of 'helpfulness to achieve learning 
outcome' for competence keywords were significantly better than those for subject keywords. The mean 
ratings of links for the two types of keywords did not differ according to type of competence node; the two 
types of keywords gave similar results. While it is commonly thought that the Web is a sufficient resource for 
learning and learners, it was expected that the Web would support the competence keywords for better 
results. The obtained results, however, showed no significant differences between the two types of keyword. 
All the generated links from both subject and competence keywords were reviewed and it was confirmed that 
they mostly contained subject matter explanation, with little capability and context elements. It seems that 
the Web does not, in general, provide pedagogically-informed learning materials. It may be that future work 
should explore the value of video in sites such as YouTube 1 , Yahoo! Video 2 , and Flickr 3 since such video 


1 http:// www.youtube.com/ 

2 http://screen.yahoo.com/ 

3 http://www.flickr.com/explore/video/ 
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resources are involved in many current learning systems, for example, MOOCs (Pappano 2012, Kay et al. 
2013). Choi and Johnson (2005) concluded that context-based videos in online courses have the potential to 
enhance learners' retention and motivation better than traditional text-based instruction. Hence video 
resources can be another domain to explore. 

8.3 Experiment Three 

Experiment three explored whether a COSREW learning mode was better than a freely-browsing learning 
mode, where it was expected that learners would achieve higher test scores in the COSREW learning mode, as 
illustrated in Figurel9. In such an experiment, ideally the pre-test scores for the two groups should be equal; in 
other words, the participants in the two groups should have equal initial knowledge. 


Learning JVIode 

COSREW 

Freely_Browsing 


PreTest PostTest 

Test_Type 

Figure 19: Expected Results of Profile Graph of Mean Ratings of Test Scores of Pre-Test and Post-Test for Two 
Learning Modes (Error Bars Show +- 1SE) 



Significant improvements were shown in both modes of learning. Participants in the freely-browsing learning 
mode were initially more knowledgeable than those in the COSREW learning mode group. It could be that the 
learners' existing knowledge helped them to improve their learning, leading to the finding of no significant 
interaction between learning mode and pre-test/post-test change. This also supports the view that the Web 
may not currently be pedagogically informed. 

8.4 Generalisation of results 

While the experiments involved one subject and one stage of education, their focus was on whether and how 
enhancing content keywords with competence capabilities might improve learning. In such a context, it would 
not be expected that the specifics of the subject (photosynthesis) and level of education (GCSE) would 
particularly prevent appropriate generalisation of the findings to other subjects and other educational levels. 

9 Conclusions and Future Work 

COSREW was proposed for suggesting study materials from the Web, deriving Web links from learners' 
competences. A process for designing a competence structure and presenting its corresponding mapped XML- 
schema was also proposed. Three experiments were conducted. In experiment one, the search results 
generated by Google and by the Google API were equally good in terms of perceived ratings of learning 
outcome achievement. In experiment two, the search results generated from subject keywords and from 
competence keywords were equally good in terms of perceived ratings of learning outcome achievement. In 
experiment three, learners' improvements were equal in both modes of learning (freely-browsing and 
COSREW). From experiment two and three, it may be concluded that the Web may not currently be a good 
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resource for learning or for a pedagogy-informed approach to learning, especially for COSREW. Most Web 
pages contain subject matter-based explanation and this may not be sufficient for learning where capability 
and context is important. 

For future work, related video may better support the learning of certain competences and should be included 
in the COSREW system. Second, authors should be encouraged to provide competence metadata when 
constructing Web pages. Search engines may then find more pedagogically relevant Web pages when their 
capability and context metadata is identified. Third, Linked Data principles (Berners-Lee 2006) can be applied 
by setting the Linked Data set and related metadata as the required competence information. The resulting 
RDF links ensure all Web materials with the same data set and metadata are linked to each other, providing 
the desired pedagogically-informed support for competence-based Web resources. Fourth, context 
classification using the COMBA competency model can be explored further. The literature discusses various 
aspects of context yet this concept is still not well defined. De Jong (2007) specifies context as identity, 
location, time, environment, and relation. Sampson and Fytros (2008) define context as job, occupations, 
function, life outcome, situation, and task. Zimmermann, Lorenz and Oppermann (2007) classified context 
information into five categories: individuality, activity, location, time, and relations. A well-defined and 
standard definition of context is still needed. Fifth, a competence structure repository should be considered so 
that the structures can be stored, discovered, searched, and reused. Lastly, the outcomes of experiment three 
suggested that the learning improvement of learners in the freely-browsing mode was due to the higher initial 
knowledge of participants. In order to validate this suggestion, future work could include the study of the 
impact of knowledge on learners' learning when interacting with a freely-browsing learning mode. This is to 
explore the significant differences between the learning of knowledgeable learners and that of non- 
knowledgeable learners when interacting with a freely-browsing learning mode. 
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